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Subject: Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to Support the 
Purchase of New Radiation Detection Portal Monitors Was Not Based on Available 
Performance Data and Did Not Fully Evaluate All the Monitors' Costs and Benefits 

Since the attacks of September 11, 2001, combating terrorism has been one of the 
nation's highest priorities. As part of that effort, preventing nuclear and radioactive 
material from being smuggled into the United States — perhaps to be used by terrorists in 
a nuclear weapon or in a radiological dispersal device (a "dirty bomb") — has become a 
key national security objective. The Department of Homeland Security (DHS) is 
responsible for providing radiation detection capabilities at U.S. ports-of-entry. 1 Until 
April 2005, U.S. Customs and Border Protection (CBP) under DHS managed this 
program. However, on April 15, 2005, the president directed the establishment, within 
DHS, of the Domestic Nuclear Detection Office (DNDO), whose duties include acquiring 
and supporting the deployment of radiation detection equipment. 2 CBP continues its 
traditional screening function at ports-of-entry to interdict dangerous nuclear and 
radiological materials through the use of radiation detection equipment, including portal 



'The Departments of Energy, Defense, and State are also implementing programs to combat nuclear 
smuggling in other countries by providing radiation detection equipment and training to foreign border 
security personnel. 

^National Security Presidential Directive 43/Homeland Security Presidential Directive 14, Domestic Nuclear 
Detection (April 15, 2005). Congress recently passed a bill that would provide permanent statutory 
authority for DNDO and its activities. See Security and Accountability for Every Port Act of 2006, H.R. 
4954, §§ 501-502 (2006) ("SAFE Port Act"). As of October 12, 2006, the President had not yet signed the 
bill. 
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monitors. The Pacific Northwest National Laboratory (PNNL), one of the Department of 
Energy's (DOE) national laboratories, manages the deployment of radiation portal 
monitors for DHS. 3 

Current portal monitors, which cost about $55,000 per monitor, detect the presence of 
radiation, but cannot distinguish between harmless radiological materials, such as 
naturally occurring radiological material in some ceramic tile, and dangerous nuclear 
materials, such as highly enriched uranium (HEU). CBP officers also use radioactive 
isotope identification devices (RIIDs), which are handheld devices designed to identity 
different types of radioactive material, such as radioactive material used in medicine or 
industry, a naturally occurring source of radiation, or weapons-grade material. These 
devices have limitations in their ability to detect and identify nuclear material. DHS 
would like to improve the capabilities of its radiation detection equipment in order to 
better distinguish between different types of nuclear and radiological materials. As a 
result, DHS sponsored research, development, and testing activities in 2005 that were 
designed to produce portal monitors that, in addition to detecting, would also identify 
the type of nuclear or radiological material. Portal monitors with this new identification 
technology currently cost about $377,000 or more per monitor. In these same tests, DHS 
also tested the performance of currently deployed portal monitors. 

In July 2006, DHS announced that it had awarded contracts to three vendors to further 
develop and purchase $1.2 billion worth of new portal monitors over 5 years. DHS plans 
to deploy these monitors at U.S. ports of entry. For fiscal year 2007, DNDO plans to 
acquire the first installment of 104 new portal monitors that use new identification 
technology at a cost of $80.2 million. Congress, however, has curtailed DNDO's ability to 
do so by restricting the availability of funding for full scale procurement of new radiation 
detection portal monitors until DHS certifies that a significant increase in operational 
effectiveness will be achieved. 4 

In a March 2006 report, to ensure that DHS's substantial investment in radiation 
detection technology yields the greatest possible level of detection capability at the 
lowest possible cost, we recommended that once the costs and capabilities of these new 
portal monitors were well understood, and before any of the new equipment was 
purchased, the Secretary of DHS work with the Director of DNDO to analyze the benefits 
and costs of deploying new portal monitors. 5 Further, we recommended that this 
analysis focus on determining whether any additional detection capability provided by 
the new portal monitors were worth the additional costs. 



'DOE manages the largest laboratory system of its kind in the world. The mission of DOE's 22 laboratories 
has evolved. Originally created to design and build atomic weapons, these laboratories have since 
expanded to conduct research in many disciplines-from high-energy physics to advanced computing. 
'Homeland Security Appropriations Act for Fiscal Year 2007, Pub. L. No. 109-295, tit. IV (2006). At the same 
time, Congress has directed that all containers entering the 22 busiest U.S. ports be scanned for radiation 
by December 31, 2007, and specified that to the extent practicable, DHS is to deploy "next generation 
radiation detection technology." See SAFE Port Act, § 121. 

D GAO, Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detection 
Equipment at U.S. Ports of Entry, but Concerns Remain, GAO-06-389 (Washington, D.C.: Mar. 22, 2006). 



Page 2 



GAO-07-133R DNDO's Cost-Benefit Analysis 



In response to our recommendation, in May 2006, DNDO issued a cost-benefit analysis 
for the acquisition and deployment of new portal monitors. 6 In this document, DNDO 
stated that the purpose of its analysis was to help provide a "robust" defense against 
nuclear smuggling, to limit the negative impacts to legitimate trade and travel between 
nations, and to provide a sound financial investment for the United States government. 
In this context, by agreement with your staff, we reviewed DNDO's cost-benefit analysis 
to determine the extent to which it provides a sound analytical basis for acquiring and 
deploying new portal monitors. We briefed your staff on the results of our evaluation on 
August 21, 2006, and this report presents the details of that briefing. The briefing 
focused mainly on the use of performance and cost data for current and newer portal 
monitors (see Enclosure 1). 

To conduct our review of DNDO's cost-benefit analysis we analyzed documentation 
supporting the costs used for both currently deployed and new equipment. As provided 
by federal regulations, DHS developed what is called an Independent Government Cost 
Estimate. 7 We obtained and evaluated this estimate to determine its reasonableness. 
We also reviewed test reports for new portal monitors and currently deployed monitors 
produced by National Institutes of Standards and Technology (NIST) and PNNL to assess 
the validity of performance assumptions DNDO used in its analysis. In addition, we had 
discussions with officials from DNDO, CBP, NIST, and several of DOE's national 
laboratories, including Oak Ridge, Sandia, Brookhaven, and PNNL, as well as 
representatives of the trucking and shipping industries. We conducted our review from 
May 2006 to September 2006 in accordance with generally accepted government auditing 
standards. 

Summary 

DNDO's cost-benefit analysis does not provide a sound analytical basis for DNDO's 
decision to purchase and deploy new portal monitor technology. DNDO did not use the 
results of its own performance tests in its cost-benefit analysis and instead relied on 
assumptions of the new technology's anticipated performance level. Performance tests 
also showed that the ability of new radiation detection portal monitors to correctly 
identify masked HEU (placed next to or within another, usually more benign, 
radiological substance) was even more limited. 8 According to the cost-benefit analysis 
and radiation detection experts to whom we spoke, masked HEU is a significant concern 
because it is difficult to detect. DNDO also focused the analysis exclusively on 
identifying HEU and did not consider in the analysis how well (either as a goal or in 
testing) new portal monitor technology can correctly detect or identify other dangerous 
radiological or nuclear materials. Furthermore, the analysis did not include the results 
from side-by-side tests that DNDO conducted of the advanced portal monitors and 
current portal monitors. 



6 DNDO also issued a revised analysis in June 2006. 

'Federal Acquisition Regulation, 48 C.F.R. § 15.404-l(b)(2)(v). 

8 These test results involved identifying both HEU and depleted uranium. 
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The cost-benefit analysis for acquiring and deploying portal monitors is also incomplete 
because it does not include all of the major costs and benefits required by DHS 
guidelines. In particular, DNDO did not assess the likelihood that radiation detection 
equipment would either misidentify or fail to detect nuclear or radiological material. 
Rather, it focused its analysis on reducing the time necessary to screen traffic at border 
check points and reduce the impact of any delays on commerce. DNDO also used 
questionable assumptions about the procurement costs of portal monitor technology. 
DNDO assumed a purchase price for current portal monitor technology that is more than 
twice what CBP typically pays. 

We provided DHS with a draft of this report for its review and comment. Its written 
comments are presented in Enclosure 2. The Department neither agreed nor disagreed 
with our recommendation, but continues to stand behind the basic conclusions of its 
cost-benefit analysis that new radiation detection portal monitors are a sound investment 
for the U.S. Government. We continue to believe, however, that significant weaknesses 
in DNDO's cost-benefit analysis precludes it from providing a reliable basis for making 
major procurement decisions, such as whether to invest heavily in deploying a new 
portal monitor technology. 

Background 

DHS's program to deploy radiation detection equipment at U.S. ports-of-entry has two 
goals. 9 The first is to use this equipment to screen all cargo, vehicles, and individuals 
coming into the United States. Each day, DHS processes about 64,000 containers 
arriving in the United States via ships, trucks, and rail cars; 365,000 vehicles; and more 
than 1.1 million people. The United States has over 380 border sites at which DHS plans 
to deploy radiation detection equipment. The second goal of the program is to screen all 
of this traffic without delaying its movement into the nation. To illustrate the difficulty of 
achieving this second goal, CBP's port director at the San Ysidro, California, land border 
crossing estimated that, prior to initiating radiation detection screening, the volume of 
traffic through the port-of-entry was so great that, at times, the wait to enter the United 
States from Mexico was about 2.5 hours. He noted that had radiation detection screening 
added a mere 20 seconds to the wait of each vehicle, the wait during those peak times 
could have increased to about 3.5 or 4 hours — an unacceptable outcome in his view. 
DHS's current plans call for completing deployments of radiation portal monitors at U.S. 
ports-of-entry by September 2009. 

To screen commerce for radiation, CBP uses several types of detection equipment and a 
system of standard operating procedures. Current radiation detection equipment 
includes radiation portal monitors, which can detect gamma radiation (emitted by 
radiological materials that can be used in a dirty bomb as well as naturally occurring 
radiological material in some ceramic tile) and neutron radiation (emitted by only a 
limited number of materials, such as plutonium — a material that can be used to make a 
nuclear weapon). 



"These goals are now subject to the new requirements established by Congress in section 121 of the SAFE 
Ports Act. 
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Generally, CBP's standard procedures direct vehicles, containers, and people coming 
into the country to pass through portal monitors to screen for the presence of radiation. 
This "primary inspection" serves to alert CBP officers that a radioactive threat might be 
present. Because current portal monitors using detection technology — known as 
"plastic scintillators" (PVT) — detect the presence of radiation but cannot distinguish 
between harmless and dangerous nuclear or radiological materials, "secondary 
inspections" are required. To confirm and identify the presence of radiation, this 
secondary inspection includes CBP officers using RIIDs to determine whether the 
radiation being emitted is from a harmless source, such as kitty litter, or a dangerous 
source, such as weapons-grade nuclear material. Typically, completing a secondary 
inspection takes about 15 minutes. DHS would like to improve the capabilities of its 
radiation detection equipment in order to distinguish among different types of nuclear 
and radiological materials and, in so doing, reduce the number of secondary inspections 
on non-threatening nuclear and radiological material. 10 

DNDO Relied on the Potential Performance of New Portal Monitors Rather Than 
Its Own Test Results of Current Equipment Performance in Its Cost-Benefit 
Analysis 

DHS is developing new portal monitors, known as "advanced spectroscopic portal 
monitors" (ASP) that, in addition to detecting nuclear or radiological material, can also 
identify the type of material. In 2005, DNDO conducted side-by-side testing at the 
Nevada Test Site (NTS) 11 on 10 ASP systems and 3 PVT systems developed by private 
sector companies, including the PVT systems currently deployed. DHS requested that 
NIST provide assistance by conducting an independent analysis of data acquired during 
the last phase of developmental testing of ASPs to help DHS determine the performance 
of ASP portal monitors being proposed by private sector companies. NIST compared the 
10 ASP systems, and in June 2006 submitted a report to DHS on the results of that 
testing. 12 

Performance tests of ASPs showed that they did not meet DNDO's main performance 
assumption in the cost-benefit analysis of correctly identifying HEU 95 percent of the 
time it passes through portal monitors. The 95 percent performance assumption 
included ASPs' ability to both detect bare, or unmasked, HEU in a container and HEU 
masked in a container with a more benign radiological material. 1J Based on NIST's 
assessment of the performance data, the ASP prototypes (manufactured by the three 
companies that won DNDO's recent ASP procurement contract) tested at NTS identified 



'"Section 121 of the SAFE Ports Act reinforces this goal by requiring that DHS utilize next generation 
radiation technology "to the extent practicable", with conference report language noting that "such 
technology can reduce nuisance alarms" and "speed effective alarm resolution." See H.R. Conf. Rep. No. 
109-711 (2006). 

"DHS and DOE are collaborating in building a new Radiological and Nuclear Countermeasures Test and 
Evaluation Complex at the Nevada Test Site to support the development, testing, acquisition, and 
deployment of radiation detection equipment. The facility is expected to become fully operational in early 
2007. Currently, an interim facility at NTS is being used to test radiation detection equipment. 
12 NIST did not evaluate the PVTs or compare their performance to the performance of the ASPs. 
13 The ability to detect masked HEU is based on DOE guidance on performing the evaluation. 
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bare HEU only 70 to 88 percent of time. Performance tests also showed that ASPs' 
ability to identify masked HEU fell far short of meeting the 95 percent level of 
performance assumed in the cost-benefit analysis. According to DNDO's cost-benefit 
analysis, identifying masked HEU is the most difficult case to address. DOE officials told 
us that benign radiological materials could be used to hide the presence of HEU. NIST 
reported that the best ASP prototype DNDO tested last year in Nevada that won a 
procurement contract was able to correctly identify masked HEU and depleted uranium 
(DU) only 53 percent of the time. Similarly, the ASP prototypes submitted by the other 
two companies that won DNDO's ASP procurement were able to identify masked HEU 
and DU only 45 percent and 17 percent of the time. 

Despite these results, DNDO did not use the information from these tests in its cost- 
benefit analysis. Instead, DNDO officials told us that since new portal monitors cannot 
meet the 95 percent level of performance, they relied on the assumption that they will 
reach that level of performance sometime in the future. DNDO officials asserted that the 
current performance levels of the ASPs will improve, but they provided no additional 
information as to how the 95 percent performance goal will be achieved or an estimate of 
when the technology will attain this level. 

Moreover, DNDO's cost-benefit analysis only considered the benefits of ASPs' ability to 
detect and identify HEU and did not consider ASPs ability to detect and identify other 
nuclear and radiological materials. The ability of an ASP to identify specific nuclear or 
radiological materials depends on whether the ASP contains software that is specific to 
those materials. In our view, a complete cost-benefit analysis would include an 
assessment of ASPs' ability to detect and identify a variety of nuclear and radiological 
material, not just HEU. By excluding radiological and nuclear materials other than HEU, 
DNDO's analysis did not consider the number of secondary inspections that would be 
related to these materials and hence it likely underestimated the costs of ASP use. 
Further, DNDO told us the assumptions for the ability of ASP systems to detect and 
identify HEU 95 percent came from the ASP performance specifications. However, we 
examined the performance specifications and found no specific requirement for 
detecting or identifying HEU with a 95 percent probability. 14 While there is a requirement 
in the performance specification for the identification for HEU and other special nuclear 
material, we found no associated probability of success in performing this function. 

DNDO's cost-benefit analysis also may not accurately reflect the capabilities of PVTs to 
detect nuclear or radiological material. DNDO officials acknowledged that DNDO tested 
the performance of PVTs along with the ASPs in 2005, but did not use the results of these 
tests in its cost-benefit analysis. Instead, in a June 2006 written response to our 
questions, DNDO told us it based its analysis on the performance of PVT monitors that 
PNNL tested during 2004 in New York. However, the results from these field tests are 
not definitive because, as PNNL noted in its final report, the tests did not use HEU and, 
therefore, the results from the tests did not indicate how well PVT portal monitors would 



"The performance specifications contain a requirement for detecting, not identifying, californium-252 with 
a 95 percent probability. Californium-252 has similar radiological properties to HEU. In addition, the 
performance specifications contain a requirement for detecting, but not identifying, other radiological 
materials such as cobalt-57, cobalt-60, barium-133, cesium-137, and americium-241. 
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be able to detect HEU in the field. Moreover, the PVT portal monitors that PNNL used 
had only one radiation detection panel as opposed to the four-panel PVT monitors that 
DHS currently deploys at U.S. ports-of-entry. An expert at a national laboratory told us 
that larger surface areas are more likely to detect radiological or nuclear material. DNDO 
also stated that due to the nature of the testing at the Nevada Test Site, the tests did not 
provide the data needed for the cost-benefit analysis. According to DNDO officials, this 
data would come from analysis of the performance of fielded systems at U.S. ports-of- 
entry where the probability to detect threats could be compared to false alarm rates. 

DNDO's Cost-Benefit Analysis was Incomplete and Used Inflated Cost Estimates 
for PVT Equipment 

DNDO officials told us they did not follow the DHS guidelines for performing cost- 
benefit analyses in conducting their own cost-benefit analysis. 16 DNDO officials said 
their analysis was intended to be an easy-to-understand assessment of the best 
investment for the U.S. government. However, DNDO's simplified approach to the cost- 
benefit analysis omitted many factors that could affect the cost of new radiation portal 
monitors. For example, DNDO officials told us that there are over 12 different types of 
ASP monitors, yet they only estimated the cost of cargo portal monitors that would be 
used at land border crossings. In reality, DNDO and CBP plan to deploy different types 
of ASPs that would have varying costs, such as portal monitors at seaports which would 
have higher costs. Additionally, DNDO did not capture all the costs related to developing 
the different types of ASP monitors. 

DNDO has also underestimated the life-cycle costs for operations and maintenance for 
both PVT and ASP equipment over time. DNDO analysis assumed a 5-year life-cycle for 
both PVT and ASP equipment. However, DNDO officials told us that a 10-year life-cycle 
was a more reasonable expectation for PVT and ASP equipment. The analysis assumes 
that the annual maintenance costs for PVT and ASP monitors will each equal 10 percent 
of their respective procurement costs. This means that maintenance costs for PVTs 
would be about $5,500 per year per unit based on a $55,000 purchase price and ASP 
maintenance costs would be about $38,000 per year per unit based on a $377,000 
purchase price. With the much higher maintenance costs for ASPs and doubling the life- 
cycle to 10 years, the long-term implications for these cost differences would be 
magnified. As a result of this, DNDO's analysis has not accounted for about $181 million 
in potential maintenance costs for ASPs alone. 

Furthermore, DNDO did not assess the likelihood that radiation detection equipment 
would either misidentify or fail to detect nuclear and radiological materials. Rather, 
DNDO's cost-benefit analysis focuses on the ability of ASPs to reduce false alarms — 
alarms that indicate nuclear or radiological material is in a container when, in fact, the 
material is actually non-threatening, such as ceramic tile. Reducing false alarms would 



Pacific Northwest National Laboratory, A Sensitivity Comparison ofNal and PVT Portal Monitors at a 
Land-Border Port-of -Entry, p. hi, November 2004. For Official Use Only 

16 DHS, Capital Planning and Investment Control, Cost-Benefit Analysis (CBA) Guidebook 2006, Version 
2.0, February 2006. Traditional rules of performing cost-benefit analyses include assessing the full life- 
cycle costs for operation and maintenance, and determining the level of confidence in cost data. 
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reduce the number of secondary inspections of non-threatening nuclear and radiological 
materials and therefore the costs of those inspections. However, as required by DHS's 
guide to performing cost-benefit analyses, DNDO's analysis does not include all costs. In 
particular, the analysis does not include the potentially much bigger cost of "false 
negatives." False negatives are instances in which a container possesses a threatening 
nuclear or radiological material, but the portal monitor either misidentifies the material 
as non-threatening or does not detect the material at all, thus allowing the material to 
enter the country. During the 2005 Nevada tests, the incidence of false negatives among 
the three vendors who received contracts ranged from about 45 percent to slightly more 
that 80 percent. This raises concerns because, as explained to us by a scientist at a 
national laboratory, at this level of performance, ASPs could conceivably misidentify 
HEU as a benign nuclear or radiological material or not detect it at all, particularly if the 
HEU is placed side by side with a non-threatening material such as kitty litter. 

In addition, in recent testimonies before Congress, DNDO's Director has cited the 
primary benefit of deploying ASP monitors as reducing unnecessary secondary 
inspections. 17 However, DNDO's analysis does not fully estimate today's baseline costs 
for secondary inspections, which makes it impossible to determine whether the use of 
ASPs, as currently planned, will result in significant cost savings for these inspections. 
Finally, the ASP contract award has exceeded DNDO's estimate for total cost by about 
$200 million. The cost-benefit analysis shows the total cost for deploying PVT and ASP 
monitors to be about $1 billion, which covers all costs related to acquisition (for both 
PVT and ASP), design, maintenance and physical inspection over 5 years. However, in 
July 2006, DHS announced that it had awarded contracts to develop and purchase up to 
$1.2 billion worth of ASP portal monitors over 5 years. Furthermore, the cost-benefit 
analysis underestimates total deployment costs and does not account for other major 
costs, such as physical inspections of cargo containers, an additional procurement of 442 
new PVT monitors, installation and integration, and maintenance. 

DNDO's Cost-Benefit Analysis Used Inflated Cost Estimates for PVT Equipment 

DNDO overstated the purchase price of PVT monitors. Although DHS is currently paying 
an average of about $55,000 per monitor, DNDO's cost-benefit analysis assumed the PVT 
would cost $130,959 — the highest published manufacturers' price for the government. 18 
According to DNDO's Director, DNDO chose the highest published price because the 
current contract for portal monitors is going to expire, and the portal monitors will 
probably cost more in the future. However, the information DNDO provided us does not 
explain why DNDO assumes that this future price will be more than double what DHS is 
currently paying, as assumed in DNDO's analysis. PNNL officials told us that the future 
price will almost certainly be lower than the price used in DNDO's analysis. 



1 'Enlisting Foreign Cooperation in U.S. Efforts to Prevent Nuclear Smuggling: Hearing Before the House 
Committee on Homeland Security, Subcommittee on Prevention of Nuclear and Biological Attack, 109 th 
Cong. (May 25, 2006) (statement of Mr. Vayl S. Oxford, Director, DNDO); Detecting Smuggled Nuclear 
Weapons, Hearing Before the Senate Judiciary Committee, Subcommittee on Terrorism, Technology, 
and Homeland Security, 109 th Cong. (July 27, 2006) (statement of Mr. Vayl S. Oxford, Director, DNDO). 
18 DNDO, Cost Benefit Analysis for Next Generation Passive Radiation Detection of Cargo an the 
Nation's Border Crossings, May 30, 2006. 
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Conclusions 



DNDO's cost-benefit analysis does not justify its recent decision to spend $1.2 billion to 
purchase and deploy ASP technology. In particular, DNDO used incomplete and 
unreliable data to evaluate the costs and benefits of ASPs. For example, DNDO did not 
use its own test results that showed that the new portal monitor technology could not 
meet the cost-benefit analysis' main performance assumption of correctly identifying 
HEU 95 percent of the time. In addition, the ability of ASPs to correctly identify masked 
HEU falls far short of the 95 percent level of performance. Instead of relying on 
performance data, DNDO relied on potential future performance to justify the purchase 
of ASPs. While DNDO officials asserted that the current performance levels of the ASPs 
will improve, they provided no additional information as to how or when the 95 percent 
performance goal will be achieved. Furthermore, DNDO's simplified approach to the 
cost-benefit analysis omitted many factors that could affect the cost of new radiation 
portal monitors, such as underestimating the life-cycle costs for operating and 
maintaining the equipment over time by about $181 million. For these reasons, DNDO's 
cost-benefit analysis does not meet the intent of our March 2006 report recommendation 
to fully assess the costs and benefits before purchasing any new equipment. 

Recommendations for Executive Action 

We recommend that DHS and DNDO conduct a new cost-benefit analysis to justify the 
purchase of new radiation detection portal monitors by using sound analytical methods, 
including actual performance data and a complete accounting of all major costs and 
benefits as required by DHS guidelines. We also recommend that DNDO conduct 
realistic testing for both ASPs and PVTs in order to meet the Congressional requirement 
outlined in the Homeland Security Appropriations Act for Fiscal Year 2007 that DHS 
certify that a "significant increase in operational effectiveness" will be achieved by ASPs 
before spending additional funds to purchase ASPs for deployment. 

Agency Comments and Our Evaluation 

We provided DHS with a draft of this report for its review and comment. Its written 
comments are presented in Enclosure 2. The Department neither agreed nor disagreed 
with our recommendation, but continues to stand behind the basic conclusions of its 
cost-benefit analysis that the ASP program is a sound investment for the U.S. 
Government. In our view, the cost-benefit analysis cannot be used as a reliable basis for 
making a major procurement decision. 

In its written comments, the Department stated that there are three areas of 
disagreement and/or misunderstanding with GAO. In our view, however, there are no 
misunderstandings on any of these issues. First, although the Department agreed that 
performance tests of ASPs did not meet DNDO's main performance assumption of 
correctly identifying HEU 95 percent of the time it passes through the portal monitors, 
these tests were designed to determine which contractors would manufacture the 
equipment, not determine the absolute capabilities of ASPs. However, in our view, an 
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objective and complete cost-benefit analysis should have included empirical data on how 
well ASPs are currently performing. Instead of relying on performance data, DNDO 
relied on potential future performance to justify the purchase of ASPs. 

Second, according to the Department, DNDO did not use performance data of PVTs in its 
cost-benefit analysis from tests conducted at the Nevada Test Site in 2005 because those 
tests were not designed to provide an objective side-by-side comparison of ASPs and 
PVTs. However, the purpose of the cost-benefit analysis was to determine the benefits, if 
any, of using ASPs instead of currently deployed radiation detection technology, 
including PVTs. The relative capabilities of the two competing detection systems — ASPs 
and PVTs — should have been critical to DNDO's analysis. However, DNDO missed an 
opportunity to collect objective, empirical data on the relative capabilities of ASPs and 
PVTs because the agency failed to design its side-by-side tests to measure such 
differences. As a result, instead of having empirical data available for its cost-benefit 
analysis, DNDO relied on assumptions of future ASP performance and on limited, 
unreliable data drawn from tests conducted in 2004 to assess the performance of PVTs. 
DNDO will not be able to assess objectively whether ASPs offer any advantages over 
PVTs, much less whether any such potential advantages are worth the ASPs' 
considerable additional cost, until DNDO conclusively determines the relative 
effectiveness of the two technologies. 

Lastly, while the Department agreed that radiation detection portal monitors should also 
be able to detect and identify radiological materials in addition to HEU, it believes that 
HEU is the most difficult material to detect and it will continue use HEU as a reasonable 
threat baseline. We agree that HEU is the most difficult material to detect, but we 
disagree that HEU should be used as the only reasonable threat baseline to determine 
the performance of portal monitors. As we reported, a complete cost-benefit analysis 
would include an assessment of ASPs' ability to detect and identify a variety of nuclear 
and radiological material, not just HEU. By excluding radiological and nuclear materials 
other than HEU, DNDO's analysis did not consider the number of secondary inspections 
that would be related to these materials and hence it likely underestimated the costs of 
ASP use. 



We are sending copies of this correspondence to interested congressional committees 
and members, the Secretary of Homeland Security, and other interested parties. We will 
also make copies available to others on request. In addition, this correspondence will be 
available at no charge on GAO's Web site at http://www.gao.gov. Should you or your 
staff have any questions, please contact me at (202) 512-6870 or by e-mail at 
aloisee@gao.gov. Contact points for our Office of Congressional Relations and Public 
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Affairs may be found on the last page of this correspondence. Key contributions to this 
report include Leland Cogliani, Nancy Crothers, Jonathan Fremont, Jim Shafer, Daren 
Sweeney, and Eugene Wisnoski. 




Gene Aloise 

Director, Natural Resources 
and Environment 
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Introduction 



> The Domestic Nuclear Detection Office (DNDO), within the 
Department of Homeland Security (DHS), is responsible for 
acquiring and supporting the deployment of radiation detection 
equipment at U.S. ports-of-entry. 

> The purpose of the radiation detection equipment is to screen all 
cargo, vehicles, and individuals coming into the United States for 
radiological and nuclear materials, without delaying movement of 
this traffic into the nation. 

> Current plastic portal monitors (PVT) are able to detect the 
presence of radiation. DNDO has sponsored research, 
development, and testing activities designed to produce portal 
monitors that can detect radiation and identify the type of 
material. These new portal monitors are known as "advanced 
spectroscopic portals" (ASP). 
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Introduction 



> In our March 2006 report 1 , we recommended that once the costs 
and capabilities of ASPs were well understood, and before any of 
the new equipment was purchased, the Secretary of DHS work 
with the Director of DNDO to analyze the benefits and costs of 
deploying ASPs. 

> We further recommended that this analysis focus on determining 
whether any additional detection capability provided by the ASPs 
are worth the additional costs. 

> In response to GAO's recommendation, in May 2006, DNDO 
issued a cost-benefit analysis to help guide its strategy for the 
acquisition and deployment of portal monitors. 

See GAO, Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detection Equipment at U.S. Ports of Entry, but Concerns Remain, GAO- 
06-389 (Washington, D.C.: Mar 22, 2006). 
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Introduction 



DNDO's Cost-Benefit Analysis: Purpose and Goals 

> The purpose of the analysis was to identify the best strategy that 
the DNDO-Customs and Border Protection could use to meet 
their deployment requirements for nuclear and radiological 
detection equipment at U.S. ports of entry. 

> The analysis had three g oals : to (1) provide a robust defense 
against nuclear and radiological threats, (2) limit impacts to 
commerce, and (3) provide a sound financial investment for the 
U.S government. 
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>The House and Senate Appropriations Committees asked 
us to review DNDO's cost-benefit analysis to determine the 
extent to which it provides a sound analytical basis for 
acquiring and deploying portal monitors. 
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Scope and Methodology 



>To conduct our review of DNDO's cost-benefit analysis, we: 

> Analyzed documentation supporting the costs used for 
both currently deployed and new equipment, such as the 
Independent Government Cost Estimate for ASPs. 

> Reviewed government test reports for ASP and PVT 
systems to assess the validity of assumptions DNDO 
used in its analysis. 

> Had discussions with officials from DNDO, Customs and 
Border Protection, Department of Energy laboratories, 
National Institutes of Standards and Technology, and 
representatives of the trucking and shipping industries. 
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DNDO Relied on the Potential Performance of New 
Portal Monitors Rather Than Its Own Test Results in Its 

Cost-Benefit Analysis 



> DNDO assumed in its cost-benefit analysis that ASPs can 
correctly detect and identify highly enriched uranium (HEU) 
95 percent of the time it passes through portal monitors. 

> In 2005, DNDO tested ASP capabilities to identify both bare, 
or unmasked, HEU and HEU masked in a container with 
radiological material, such as kitty litter. 

> Performance tests showed that ASPs identified HEU much 
less than 95 percent of the time. At best, the tested ASPs 
correctly identified masked HEU only about half of the time. 

> DNDO did not use the results of these tests in its cost- 
benefit analysis and instead assumed that the ASPs will 
reach that level of performance in the future. 
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DNDO's Analysis May Not Accurately Reflect the 
Capabilities of PVTs to Detect Radiological Materials 



> DNDO officials acknowledged that DNDO tested the 
performance of PVTs along with the ASPs in 2005, but did 
not use the results of these tests in its cost-benefit analysis. 

> Instead, DNDO based its analysis on the performance of 
PVT monitors that PNNL tested during 2004 in New York 

>The PVT monitors that DHS used in the tests had only one 
radiation detection panel as opposed to the four-panel PVT 
monitors that DHS currently deploys at ports-of-entry. 

> Additionally, PNNL noted that the results from the tests did 
not indicate how well PVT portal monitors would be able to 
detect HEU in the field. 
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DNDO's Cost-Benefit Analysis was Incomplete 



> DNDO officials told us they did not follow DHS guidelines for 
performing cost-benefit analyses. 

> DNDO's cost-benefit analysis only considered the benefits of 
ASPs ability to detect and identify HEU and did not consider 
ASPs ability to detect and identify other nuclear and radiological 
materials. 

> A complete analysis would include an assessment of an ASP's 
ability to identify a variety of radiological and nuclear sources, not 
just HEU. 

> DNDO's analysis does not include the bigger cost of instances in 
which a container possesses a threatening radiological or nuclear 
material, but the portal monitor either misidentifies the material as 
non-threatening or does not detect the material at all, thus 
allowing the material to enter the country. 
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DNDO's Cost-Benefit Analysis was Incomplete 



> DNDO analysis assumed a 5-year life-cycle for both PVT 
and ASP equipment. However, DNDO officials told us that 
a 10-year life-cycle was a more reasonable expectation for 
PVT and ASP equipment. 

> Maintenance costs for PVTs would be about $5,500 per 
year per unit based on a $55,000 purchase price and ASP 
maintenance costs would be about $38,000 per year per 
unit based on a $377,000 purchase price. 

>With the much higher maintenance costs for ASPs and 
doubling the life-cycle to 10 years, the long-term 
implications for these cost differences would be magnified. 
As a result of this, DNDO's analysis has not accounted for 
about $181 million in maintenance costs for ASPs alone. 
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>ln calculating costs and benefits, DNDO overstated 
the acquisition costs of PVTs. 

>DNDO assumed PVTs would cost about $131 ,000 
per unit — the highest published manufacturers' 
price for the government. 

>However, DHS's actual procurement cost for a PVT 
monitor currently averages about $55,000. 
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• DNDO's analysis shows the total cost for deploying PVT 
and ASP monitors to be about $1 billion; which covers all 
costs related to acquisition (for both PVT and ASP), design, 
maintenance and physical inspection over 5 years. 

• However, in July 2006, DHS announced that it had awarded 
contracts to develop and purchase $1 .2 billion worth of ASP 
portal monitors over 5 years. 

• The ASP contract award has exceeded DNDO's estimate 
for total cost by about $200 million. This is before 
accounting for major aspects of the total cost, such as the 
costs of physical inspections of cargo, an additional 
procurement of 442 new PVT monitors, installation and 
integration costs, and maintenance costs. 
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GAO's Conclusions and Recommendations 



> DNDO's analysis is incomplete and unreliable. 

> DNDO did not use its own test results and instead assumed 
relied on potential performance of the new equipment. 

> DNDO used inaccurate or incomplete cost data. 

> The analysis does not include all costs and benefits for 
acquiring and deploying portal monitors 

> Thus, DNDO's cost benefit analysis does not justify its recent 
decision to purchase and deploy ASP technology. 

> DNDO should conduct a new cost-benefit analysis to justify the 
purchase of new radiation detection portal monitors by using 
sound analytical methods. 

> DNDO should not spend any additional funds to purchase ASPs 
for deployment until it conducts realistic testing for both ASPs and 
PVTs that demonstrates that the new equipment provides 
superior detection and identification capabilities when compared 
to existing technology. 
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Comments from the Department of Homeland Security 



Domestic Nuclear Detection Office 

US. Department of Homeland Security 

Washington, DC 20528 




Homeland 
Security 



October 11,2006 



MEMORANDUM FOR: 



Gene Aloise 

Director, Natural Resources and Environment 
Government Accountability Office 



FROM: 




Director 

Domestic Nuclear Detection Office 

Comments on GAO Report titled: Combating Nuclear Smuggling: 
DHS's Cost-Benefit Analysis to Support the Purchase of New 
Radiation Detection Portal Monitors Was Not Based on Available 
Performance Data and Did Not Fully Evaluate All the Monitors' Costs 
and Benefits, dated October 12, 2006 



The Domestic Nuclear Detection Office (DNDO) has reviewed the subject report and the briefing 
presented to the House and Senate Committees on Appropriations dated August 21, 2006. Thank you 
very much for the opportunity to comment. While the DNDO has concerns with some of the 
report's findings, we remain committed to high fidelity testing prior to all full scale acquisition 
decisions, including the Advanced Spectroscopic Portal (ASP) program. 

The DNDO met on multiple occasions with the GAO staff to discuss the Cost Benefit Analysis 
(CBA) methodology, assumptions, data sources, and results. In all instances, we agreed with GAO's 
suggestions and worked to further refine the CBA, or provided written responses documenting the 
technical rationale for the DNDO's approach. However, some misunderstandings and/or 
disagreements remain between the DNDO and the GAO, including: 

• The GAO criticized the DNDO for assuming a probability of detection of 95%, even though 
initial test results did not show this same capability. However, the initial test results cited by 
the GAO were not intended to determine the absolute capabilities of deployed systems; 
rather, they were intended to support initial source selection decisions. Again, we remain 
committed to high fidelity testing prior to all full scale acquisition decisions. We plan to 
subject systems to additional testing prior to full scale production to evaluate their 
performance against stated specifications, including the requirement for 95% probability of 
detection. 

• The GAO stated that the DNDO tested the performance of PVT and ASP systems side-by- 
side, but did not use these results in the CBA. Again, the test series referenced was not 
intended to provide an objective side-by-side comparison. Further testing at both the Nevada 
Test Site and the New York Container Terminal will evaluate the validity of these 
assumptions. 
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• The GAO stated that the CBA only evaluated systems' ability to detect highly enriched 
uranium (HEU) and did not consider other threats. DNDO agrees that threats other than 
HEU are equally important. However, we believe that HEU poses the greatest challenge 
from a detection standpoint, and serves as a reasonable threat baseline for the purpose of this 
analysis. 

The DNDO stands behind the basic conclusions of the CBA-that the Advanced Spectroscopic Portal 
(ASP) program is a sound investment for the USG and that the Joint DNDO-CBP Deployment 
Strategy, employing a mix of PVT and ASP systems for primary inspection and ASP systems for all 
secondary inspections is the optimal approach. 

DNDO believes that this report requires further collaboration and we look forward to continuing to 
work together on this important national priority. Please let me know if you or your staff need 
additional information; (202) 254-5688. 



Attachments: 



(360691) 
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